Ensemble Techniques - Travel Package Purchase Prediction¶
Description¶
Background and Context¶
You are a Data Scientist for a tourism company named "Visit with us". The Policy Maker of the company wants to enable and establish a viable business model to expand the customer base.
A viable business model is a central concept that helps you to understand the existing ways of doing the business and how to change the ways for the benefit of the tourism sector.
One of the ways to expand the customer base is to introduce a new offering of packages.
Currently, there are 5 types of packages the company is offering - Basic, Standard, Deluxe, Super Deluxe, King. Looking at the data of the last year, we observed that 18% of the customers purchased the packages.
However, the marketing cost was quite high because customers were contacted at random without looking at the available information.
The company is now planning to launch a new product i.e. Wellness Tourism Package. Wellness Tourism is defined as Travel that allows the traveler to maintain, enhance or kick-start a healthy lifestyle, and support or increase one's sense of well-being.
However, this time company wants to harness the available data of existing and potential customers to make the marketing expenditure more efficient.
You as a Data Scientist at "Visit with us" travel company have to analyze the customers' data and information to provide recommendations to the Policy Maker and Marketing Team and also build a model to predict the potential customer who is going to purchase the newly introduced package.
Objective¶
- To predict which customer is more likely to purchase the newly introduced travel package.
Data Dictionary¶
Customer details:¶
- CustomerID: Unique customer ID
- ProdTaken: Product taken flag
- Age: Age of customer
- PreferredLoginDevice: Preferred login device of the customer in last month
- CityTier: City tier
- Occupation: Occupation of customer
- Gender: Gender of customer
- NumberOfPersonVisited: Total number of person came with customer
- PreferredPropertyStar: Preferred hotel property rating by customer
- MaritalStatus: Marital status of customer
- NumberOfTrips: Average number of the trip in a year by customer
- Passport: Customer passport flag
- OwnCar: Customers owns a car flag
- NumberOfChildrenVisited: Total number of children visit with customer
- Designation: Designation of the customer in the current organization
- MonthlyIncome: Gross monthly income of the customer
Customer interaction data:¶
- PitchSatisfactionScore: Sales pitch satisfactory score
- ProductPitched: Product pitched by a salesperson
- NumberOfFollowups: Total number of follow up has been done by sales person after sales pitch
- DurationOfPitch: Duration of the pitch by a salesman to customer
Criteria Perform an Exploratory Data Analysis on the data¶
- Univariate analysis - Bivariate analysis - Use appropriate visualizations to identify the patterns and insights - Come up with a customer profile (characteristics of a customer) of the different packages - Any other exploratory deep dive
Points : 7.5¶
Criteria Illustrate the insights based on EDA¶
Key meaningful observations on the relationship between variables
Points : 5¶
Criteria Data Pre-processing¶
Prepare the data for analysis - Missing value Treatment, Outlier Detection(treat, if needed- why or why not ), Feature Engineering, Prepare data for modeling
Points : 7.5¶
Criteria Model building - Bagging¶
- Build bagging classifier, random forest and decision tree. ### Points : 6
Criteria Model performance evaluation and improvement¶
- Comment on which metric is right for model performance evaluation and why? - Comment on model performance - Can model performance be improved? check and comment ### Points : 9
Criteria Model building - Boosting¶
- Build Adaboost, gradient boost, xgboost and stacking classifier ### Points : 8
Criteria Model performance evaluation and improvement¶
- Comment on which metric is right for model performance evaluation and why? - Comment on model performance - Can model performance be improved? check and comment ### Points : 12
Criteria Actionable Insights & Recommendations
- Compare models - Business recommendations and insights ### Points : 5
Total Points : 60¶
Overview of the dataset¶
Let's start by importing libraries we need.
In [1]:
import warnings
warnings.filterwarnings("ignore")
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor,RandomForestRegressor, GradientBoostingRegressor, AdaBoostRegressor, StackingRegressor
from xgboost import XGBRegressor
from sklearn import metrics
from sklearn.model_selection import GridSearchCV, train_test_split
In [2]:
#Loading dataset
# data = pd.read_excel("Tourism.xlsx", sheet_name="Tourism")
xls = pd.ExcelFile('Tourism.xlsx')
datadictonary = pd.read_excel(xls, 'Data Dict')
data = pd.read_excel(xls, 'Tourism')
View the first 5 rows of the dataset.
In [3]:
data.head()
Out[3]:
Check data types and number of non-null values for each column.
In [4]:
data.info()
In [5]:
data["PreferredLoginDevice"] = data["PreferredLoginDevice"].astype("category")
data["Occupation"] = data["Occupation"].astype("category")
data["Gender"] = data["Gender"].astype("category")
data["ProductPitched"] = data["ProductPitched"].astype("category")
data["MaritalStatus"] = data["MaritalStatus"].astype("category")
data["Designation"] = data["Designation"].astype("category")
Observation¶
- We can see that there are total 19 columns and 4,888 number of rows in the dataset.
- 13 columns' data type is either integer or float, 6 columns - 'PreferredLoginDevice, Occupation, Gender, ProductPitched, MaritalStatus, Designation' which is of object type.
- The number of non-null value of each column is equal to number of total rows in the dataset i.e. no null value. We can further confirm this using
isna()method.**
In [6]:
data.isna().sum()
Out[6]:
Observation¶
- Age, DurationOfPitch, NumberOfFollowups, PreferredPropertyStar, NumberOfTrips, NumberOfChildrenVisited, MonthlyIncome columns are having null values
In [7]:
data.median()
Out[7]:
In [8]:
# replace the missing values with median value.
# Note, we do not need to specify the column names below
# every column's missing value is replaced with that column's median respectively (axis =0 means columnwise)
data = data.fillna(data.median())
In [9]:
data.isna().sum()
Out[9]:
In [10]:
data.dtypes
Out[10]:
Summary of the dataset
In [11]:
# Summary of continuous columns
data.describe().T
Out[11]:
- DurationOfPitch, NumberOfTrips, MonthlyIncome are outliers
- Target variable seems to have skewed distribution as higher values are on the right. We will explore this further.
Univariate and Bivariate Analysis¶
To do - Identify insights if any from the distributuions.
Number of unique values in each column
In [12]:
data.shape
Out[12]:
In [13]:
data.nunique()
Out[13]:
- We can drop 'instant' column as it is an ID variable and will not add value to the model.
- We can drop 'dteday' column as it just contains dates of 731 days i.e. 2 years. This will not add value to the model.
Number of observations in each category
In [15]:
cat_cols=['ProdTaken', 'Age', 'PreferredLoginDevice', 'CityTier', 'DurationOfPitch', 'Occupation', 'Gender', 'NumberOfPersonVisited', 'NumberOfFollowups', 'ProductPitched', 'PreferredPropertyStar', 'MaritalStatus', 'NumberOfTrips', 'Passport', 'PitchSatisfactionScore', 'OwnCar', 'NumberOfChildrenVisited', 'Designation']
for column in cat_cols:
print('--**'*10)
print(data[column].value_counts())
print('**--'*10)
Summary & Observation¶
Product taken flag count: 0 - 3968 & 1 - 920
Preferred login device of customer in last month: Self Enquiry - 3444, Company Invited - 1419
- Self enquiry customers are more then compnay invited
City tier: (1 - 3190, 2 - 198, 3- 1500)
- Maximum customer are from type 1 city
Occupation of customer: (Salaried - 2368, Small Business - 2084, Large Business - 434, Free Lancer - 2)
- Occupation of maximum cutomers eirther Salaried or Small Business
Gender of customer: (Male - 2916, Female - 1817, Fe Male - 155)
- More than 50% customers are male
- 155 records data having value as "Fe Male" - it is related to data entry issue so need to update the value from "Fe Male" to "Female"
Total number of person came with customer: (1 - 39, 2 - 1418, 3 - 2402, 4 - 1026, 5 - 3)
- Customer with 1 and 5 persons are very less
Total number of follow up has been done by sales person after sales pitch: (1.0 - 176, 2.0 - 229, 3.0 - 1466, 4.0 - 2113, 5.0 - 768, 6.0 - 136)
- Maximum follow-up by sales person either 3 or 4 times
Product pitched by sales person - (Multi - 1842, Super Deluxe - 1732, Standard: 742, Deluxe - 342, King - 230)
- Multi , Super Deluxe types of production choosed by majority of customers
Preferred hotel property rating by customer: (3.0 - 3019, 4.0 - 913, 5.0 - 956)
- Majority of customer would like to stayed in 3 stars property
Marital status of customer - (Married -2340, Divorced- 950, Single- 916, Unmarried-682)
- Majority of customers are married
Average number of trip in a year by customer: (1.0 - 620, 2.0 - 1464, 3.0 - 1219, 4.0 - 478, 5.0 - 458, 6.0 - 322, 7.0 - 218, 8.0 - 105, 19.0 - 1, 20.0 - 1 , 21.0 - 1, 22.0 - 1)
- Evenly distributed. However, number of visits for 2 and 3 times are as majority.
Customer passport flag - (0 - 3466, 1 - 1422)
- Count of overseas customer are less
Sales pitch satisfactory score - (1 - 942, 2 - 586 , 3 - 1478, 4 - 912 , 5 - 970)
- Evenly distributed customer. However, sales persons with 3 is maximum
Customers owns a car flag - (1 - 3032, 0 - 1856)
- Customers owns a car as 1 are more, Howevere, It hardly matters
Total number of children v isit with customer - (0.0 - 1082, 1.0 - 2146, 2.0 - 1335, 3.0 - 325)
- Customers either wanted to visit alone or with 1 or 2 memeber are maximum
Designation of customer in current organization - (Executive - 1842, Manager - 1732, Senior Manager - 742, AVP - 342, VP - 230)
- Manily Executive and Manager are in Majority
Age of customer, Duration of pitch by a sales man to customer
Let update the value of Gender from Fe Male to Female¶
In [16]:
# data[ np.logical_and(data.Gender=='Fe Male'), ['Gender'] ] = 'Female'
data.loc[(data.Gender == 'Fe Male'),'Gender']='Female'
In [17]:
print(data['Gender'].value_counts())
EDA¶
Univariate analysis¶
Histogram Plot and Box Plt after above Observations for all features
In [18]:
sns.histplot(data['Age'])
Out[18]:
- Maximum customer are of age 35
In [19]:
sns.histplot(data['CityTier'])
Out[19]:
- Maximum Customer are from Type 1 city
In [20]:
sns.histplot(data['DurationOfPitch'])
Out[20]:
- Duration of pitch by a sales man to customer is below 40 and maximum count over 450 and under 10
In [21]:
sns.histplot(data['Occupation'])
Out[21]:
- Occupation of maximum cutomers eirther Salaried or Small Business
- Large is very as compare to Salaried or Small Business
In [22]:
sns.histplot(data['Gender'])
Out[22]:
- Male customers are more than 2500
- Female cutomers are less than 2000
In [23]:
sns.histplot(data['NumberOfPersonVisited'])
Out[23]:
- Over 2000 customers with 3 persons
- Over 1000 customers with 2 persons
- less then 1000 customers with 4 persons
- Customers with 1 persons are very less
In [24]:
sns.histplot(data['NumberOfFollowups'])
Out[24]:
- 4 times follow-up done with over 2000 customers
- 3 times follow-up done with over 1200 customers
- 2 times follow-up done with over 750 customers
- 1,2,6 times follow up with less customers
In [25]:
sns.histplot(data['ProductPitched'])
Out[25]:
- Product type "Multi" pitched over 1750 time by sales person
- Product type "Super Deluxe" pitched less than 1750 time by sales person
- Others are less than 750 time pitched sales person
In [26]:
sns.histplot(data['PreferredPropertyStar'])
Out[26]:
- Preferred hotel property rating by customer
- 3 star around 3000
- 4 * 5 star is less then 1000
In [27]:
sns.histplot(data['MaritalStatus'])
Out[27]:
- Marital status of customer
- Married over 2000
- Single, Divorced, Unmarried is less then 1000
In [28]:
sns.histplot(data['NumberOfTrips'])
Out[28]:
- Average number of trip in a year by customer
- 2 trip over 1400
- 3 trip over 1200
- 1 trip for around 600
- other less than 500
In [29]:
sns.histplot(data['PitchSatisfactionScore'])
Out[29]:
- Sales pitch satisfactory score
- 3 star for more than 1400
- others are less than 1000
In [30]:
sns.histplot(data['NumberOfChildrenVisited'])
Out[30]:
- Total number of children visit with customer
- 2000 customer visited with 1 children
- customer visited with 2 children is over 1000
- around 1000 customer visited with no children
In [31]:
sns.histplot(data['Designation'])
Out[31]:
- Designation of customer in current organization
- Executive are over 1750
- Managers are less than 1750
In [32]:
sns.boxplot(data['Age'])
Out[32]:
- All customers age is from 30 to 45.
- No outlier
In [33]:
sns.boxplot(data['DurationOfPitch'])
Out[33]:
- Duration of pitch by a sales man to customer
- have outliers
In [34]:
sns.boxplot(data['NumberOfPersonVisited'])
Out[34]:
- Total number of person came with customer - 5 outliers
In [35]:
sns.boxplot(data['NumberOfFollowups'])
Out[35]:
- Total number of follow up has been done by sales person after sales pitch
- outliers on both sides
In [36]:
sns.boxplot(data['PreferredPropertyStar'])
Out[36]:
- Preferred hotel property rating by customer
- no outlers for choosein property
In [37]:
sns.boxplot(data['NumberOfTrips'])
Out[37]:
- Average number of trip in a year by customer: outlier
In [38]:
#Top 5 highest values
data['Age'].nlargest()
Out[38]:
- Four customes of age 61
Function to create barplots for each feature
In [39]:
cat_cols=['ProdTaken', 'Age', 'PreferredLoginDevice', 'CityTier', 'DurationOfPitch', 'Occupation', 'Gender', 'NumberOfPersonVisited', 'NumberOfFollowups', 'ProductPitched', 'PreferredPropertyStar', 'MaritalStatus', 'NumberOfTrips', 'Passport', 'PitchSatisfactionScore', 'OwnCar', 'NumberOfChildrenVisited', 'Designation']
for column in cat_cols:
plt.figure(figsize = (20,15))
sns.countplot(data[column])
plt.show()
Bivariate analysis¶
In [40]:
cat_cols=['ProdTaken', 'Age', 'PreferredLoginDevice', 'CityTier', 'DurationOfPitch', 'Occupation', 'Gender', 'NumberOfPersonVisited', 'NumberOfFollowups', 'ProductPitched', 'PreferredPropertyStar', 'MaritalStatus', 'NumberOfTrips', 'Passport', 'PitchSatisfactionScore', 'OwnCar', 'NumberOfChildrenVisited', 'Designation']
for column in cat_cols:
sns.set(rc={'figure.figsize':(21,7)})
sns.catplot(x=column, y="Age", kind="swarm", data=data, height=7, aspect=3);
In [41]:
cat_cols=['ProdTaken', 'Age', 'PreferredLoginDevice', 'CityTier', 'DurationOfPitch', 'Occupation', 'Gender', 'NumberOfPersonVisited', 'NumberOfFollowups', 'ProductPitched', 'PreferredPropertyStar', 'MaritalStatus', 'NumberOfTrips', 'Passport', 'PitchSatisfactionScore', 'OwnCar', 'NumberOfChildrenVisited', 'Designation']
for column in cat_cols:
sns.set(rc={'figure.figsize':(21,7)})
sns.catplot(x=column, y="Age", data=data, kind='bar', size=6, aspect=1.5, estimator=np.mean);
In [42]:
sns.pairplot(
data,
height=4,
aspect=1
);
In [43]:
sns.set(rc={'figure.figsize':(16,10)})
sns.heatmap(data.corr(),
annot=True,
linewidths=.5,
center=0,
cbar=False,
cmap="YlGnBu")
plt.show()
Split the dataset¶
In [55]:
# Separating features and the target column
X = data.drop(['PreferredLoginDevice','Occupation','Gender','ProductPitched','MaritalStatus','Designation'], axis=1)
y = data[['ProdTaken','Age','CityTier','DurationOfPitch','NumberOfPersonVisited','NumberOfFollowups','Passport','PitchSatisfactionScore','OwnCar','NumberOfChildrenVisited']]
In [56]:
# encoding the categorical variables
x = pd.get_dummies(X, drop_first=True)
x.head()
Out[56]:
In [57]:
# Splitting the data into train and test sets in 70:30 ratio
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=1, shuffle=True)
In [58]:
X_train.shape, X_test.shape
Out[58]:
- We have 3,421 observations in the train set and 1,467 observations in the test set.
Building Models¶
- We'll fit different models on the train data and observe their performance.
- We'll try to improve that performance by tuning some hyperparameters available for that algorithm.
- We'll use GridSearchCv for hyperparameter tuning and
r_2 scoreto optimize the model. - R-square -
Coefficient of determinationis used to evaluate the performance of a regression model. It is the amount of the variation in the output dependent attribute which is predictable from the input independent variables. - Let's start by creating a function to get model scores, so that we don't have to use same codes repeatedly.
In [59]:
## Function to calculate r2_score and RMSE on train and test data
def get_model_score(model, flag=True):
'''
model : classifier to predict values of X
'''
# defining an empty list to store train and test results
score_list=[]
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)
train_r2=metrics.r2_score(y_train,pred_train)
test_r2=metrics.r2_score(y_test,pred_test)
train_rmse=np.sqrt(metrics.mean_squared_error(y_train,pred_train))
test_rmse=np.sqrt(metrics.mean_squared_error(y_test,pred_test))
#Adding all scores in the list
score_list.extend((train_r2,test_r2,train_rmse,test_rmse))
# If the flag is set to True then only the following print statements will be dispayed, the default value is True
if flag==True:
print("R-sqaure on training set : ",metrics.r2_score(y_train,pred_train))
print("R-square on test set : ",metrics.r2_score(y_test,pred_test))
print("RMSE on training set : ",np.sqrt(metrics.mean_squared_error(y_train,pred_train)))
print("RMSE on test set : ",np.sqrt(metrics.mean_squared_error(y_test,pred_test)))
# returning the list with train and test scores
return score_list
Decision Tree Model¶
In [60]:
dtree=DecisionTreeRegressor(random_state=1)
dtree.fit(X_train,y_train)
Out[60]:
In [61]:
dtree_score=get_model_score(dtree)
- Decision tree model with default parameters is overfitting the train data.
- Let's see if we can reduce overfitting and improve performance on test data by tuning hyperparameters.
Hyperparameter Tuning¶
In [62]:
# Choose the type of classifier.
dtree_tuned = DecisionTreeRegressor(random_state=1)
# Grid of parameters to choose from
parameters = {'max_depth': list(np.arange(2,20)) + [None],
'min_samples_leaf': [1, 3, 5, 7, 10],
'max_leaf_nodes' : [2, 3, 5, 10, 15] + [None],
'min_impurity_decrease': [0.001, 0.01, 0.1, 0.0]
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.r2_score)
# Run the grid search
grid_obj = GridSearchCV(dtree_tuned, parameters, scoring=scorer,cv=5)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the clf to the best combination of parameters
dtree_tuned = grid_obj.best_estimator_
# Fit the best algorithm to the data.
dtree_tuned.fit(X_train, y_train)
Out[62]:
In [63]:
dtree_tuned_score=get_model_score(dtree_tuned)
- The overfitting is reduced after hyperparameter tuning and test score has increased by approx 2%.
- RMSE is also reduced on test data and the model is generalizing better than the decision tree model with default parameters.
Plotting the feature importance of each variable
In [64]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
print(pd.DataFrame(dtree_tuned.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [65]:
feature_names = X_train.columns
importances = dtree_tuned.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
Random Forest Model¶
In [66]:
rf_estimator=RandomForestRegressor(random_state=1)
rf_estimator.fit(X_train,y_train)
Out[66]:
In [67]:
rf_estimator_score=get_model_score(rf_estimator)
- Random forest is giving good r2 score of 92% on the test data but it is slightly overfitting.
- Random forest is giving good r2 score of 44% on the test data and it is overfitting.
- Let's try to reduce this overfitting by hyperparameter tuning.
Hyperparameter Tuning¶
In [68]:
# Choose the type of classifier.
rf_tuned = RandomForestRegressor(random_state=1)
# Grid of parameters to choose from
parameters = {
'max_depth':[4, 6, 8, 10, None],
'max_features': ['sqrt','log2',None],
'n_estimators': [80, 90, 100, 110, 120]
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.r2_score)
# Run the grid search
grid_obj = GridSearchCV(rf_tuned, parameters, scoring=scorer,cv=5)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the clf to the best combination of parameters
rf_tuned = grid_obj.best_estimator_
# Fit the best algorithm to the data.
rf_tuned.fit(X_train, y_train)
Out[68]:
In [69]:
rf_tuned_score=get_model_score(rf_tuned)
- No significant change in the result. The result is almost same before or after the hyperparameter tuning.
In [70]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
print(pd.DataFrame(rf_tuned.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [71]:
feature_names = X_train.columns
importances = rf_tuned.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
- Age is the most important feature, also DurationOfPitch, for the tuned random forest model.
Boosting Models¶
AdaBoost Regressor¶
In [244]:
X_ada = data[['Age']]
y_ada = data[['CityTier','DurationOfPitch','NumberOfPersonVisited','NumberOfFollowups','Passport','PitchSatisfactionScore','OwnCar','NumberOfChildrenVisited']]
# X_ada = data[['Occupation','Gender','ProductPitched','MaritalStatus','Designation']]
# y_ada = data[['ProdTaken','Age','CityTier','DurationOfPitch','NumberOfPersonVisited','NumberOfFollowups','Passport','PitchSatisfactionScore','OwnCar','NumberOfChildrenVisited']]
# Separating features and the target column
# X = data.drop(['PreferredLoginDevice','Occupation','Gender','ProductPitched','MaritalStatus','Designation'], axis=1)
# y = data[['ProdTaken','Age','CityTier','DurationOfPitch','NumberOfPersonVisited','NumberOfFollowups','Passport','PitchSatisfactionScore','OwnCar','NumberOfChildrenVisited']]
In [245]:
X_train_ada, X_test_ada, y_train_ada, y_test_ada = train_test_split(X_ada, y_ada, test_size=0.30, random_state=1, shuffle=True)
In [246]:
X_train_ada
Out[246]:
In [247]:
y_train_ada
Out[247]:
In [248]:
## Function to calculate different metric scores of the model - Accuracy, Recall and Precision
def get_metrics_score(model,flag=True):
'''
model : classifier to predict values of X
'''
# defining an empty list to store train and test results
score_list=[]
pred_train_ada = model.predict(X_train_ada)
pred_test_ada = model.predict(X_test_ada)
train_acc_ada = model.score(X_train_ada,y_train_ada)
test_acc_ada = model.score(X_test_ada,y_test_ada)
train_recall_ada = metrics.recall_score(y_train_ada,pred_train_ada)
test_recall_ada = metrics.recall_score(y_test_ada,pred_test_ada)
train_precision_ada = metrics.precision_score(y_train_ada,pred_train_ada)
test_precision_ada = metrics.precision_score(y_test_ada,pred_test_ada)
score_list.extend((train_acc_ada,test_acc_ada,train_recall_ada,test_recall_ada,train_precision_ada,test_precision_ada))
# If the flag is set to True then only the following print statements will be dispayed. The default value is set to True.
if flag == True:
print("Accuracy on training set : ",model.score(X_train_ada,y_train_ada))
print("Accuracy on test set : ",model.score(X_test_ada,y_test_ada))
print("Recall on training set : ",metrics.recall_score(y_train_ada,pred_train_ada))
print("Recall on test set : ",metrics.recall_score(y_test_ada,pred_test_ada))
print("Precision on training set : ",metrics.precision_score(y_train_ada,pred_train_ada))
print("Precision on test set : ",metrics.precision_score(y_test_ada,pred_test_ada))
return score_list # returning the list with train and test scores
In [249]:
ab_regressor=AdaBoostRegressor(random_state=1)
ab_regressor.fit(X_train_ada,y_train_ada)
In [250]:
pred_train_ada = ab_regressor.predict(X_train_ada)
pred_test_ada = ab_regressor.predict(X_test_ada)
train_acc_ada = ab_regressor.score(X_train_ada,y_train_ada)
test_acc_ada = ab_regressor.score(X_test_ada,y_test_ada)
train_recall_ada = metrics.recall_score(y_train_ada,pred_train_ada)
# test_recall_ada = metrics.recall_score(y_test_ada,pred_test_ada)
# train_precision_ada = metrics.precision_score(y_train_ada,pred_train_ada)
# test_precision_ada = metrics.precision_score(y_test_ada,pred_test_ada)
In [211]:
ab_regressor_score=get_metrics_score(ab_regressor)
In [212]:
ab_regressor_score=get_model_score(ab_regressor)
Hyperparameter Tuning¶
In [183]:
# Choose the type of classifier.
ab_tuned = AdaBoostRegressor(random_state=1)
# Grid of parameters to choose from
parameters = {'n_estimators': np.arange(10,100,10),
'learning_rate': [1, 0.1, 0.5, 0.01],
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.r2_score)
# Run the grid search
grid_obj = GridSearchCV(ab_tuned, parameters, scoring=scorer,cv=5)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the clf to the best combination of parameters
ab_tuned = grid_obj.best_estimator_
# Fit the best algorithm to the data.
ab_tuned.fit(X_train, y_train)
In [ ]:
ab_tuned_score=get_model_score(ab_tuned)
In [184]:
# importance of features in the tree building
print(pd.DataFrame(ab_tuned.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [185]:
feature_names = X_train.columns
importances = ab_tuned.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
Gradient Boosting Regressor¶
In [207]:
gb_estimator=GradientBoostingRegressor(random_state=1)
gb_estimator.fit(X_train_ada,y_train_ada)
Out[207]:
In [208]:
gb_estimator_score=get_model_score(gb_estimator)
Hyperparameter Tuning¶
In [188]:
# Choose the type of classifier.
gb_tuned = GradientBoostingRegressor(random_state=1)
# Grid of parameters to choose from
parameters = {'n_estimators': np.arange(50,200,25),
'subsample':[0.7,0.8,0.9,1],
'max_features':[0.7,0.8,0.9,1],
'max_depth':[3,5,7,10]
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.r2_score)
# Run the grid search
grid_obj = GridSearchCV(gb_tuned, parameters, scoring=scorer,cv=5)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the clf to the best combination of parameters
gb_tuned = grid_obj.best_estimator_
# Fit the best algorithm to the data.
gb_tuned.fit(X_train, y_train)
In [ ]:
gb_tuned_score=get_model_score(gb_tuned)
In [ ]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
print(pd.DataFrame(gb_tuned.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [ ]:
feature_names = X_train.columns
importances = gb_tuned.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
XGBoost Regressor¶
In [189]:
xgb_estimator=XGBRegressor(random_state=1)
xgb_estimator.fit(X_train,y_train)
In [190]:
xgb_estimator_score=get_model_score(xgb_estimator)
Hyperparameter Tuning¶
In [191]:
# Choose the type of classifier.
xgb_tuned = XGBRegressor(random_state=1)
# Grid of parameters to choose from
parameters = {'n_estimators': [75,100,125,150],
'subsample':[0.7, 0.8, 0.9, 1],
'gamma':[0, 1, 3, 5],
'colsample_bytree':[0.7, 0.8, 0.9, 1],
'colsample_bylevel':[0.7, 0.8, 0.9, 1]
}
# Type of scoring used to compare parameter combinations
scorer = metrics.make_scorer(metrics.r2_score)
# Run the grid search
grid_obj = GridSearchCV(xgb_tuned, parameters, scoring=scorer,cv=5)
grid_obj = grid_obj.fit(X_train, y_train)
# Set the clf to the best combination of parameters
xgb_tuned = grid_obj.best_estimator_
# Fit the best algorithm to the data.
xgb_tuned.fit(X_train, y_train)
In [ ]:
xgb_tuned_score=get_model_score(xgb_tuned)
In [ ]:
# importance of features in the tree building ( The importance of a feature is computed as the
#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )
print(pd.DataFrame(xgb_tuned.feature_importances_, columns = ["Imp"], index = X_train.columns).sort_values(by = 'Imp', ascending = False))
In [ ]:
feature_names = X_train.columns
importances = xgb_tuned.feature_importances_
indices = np.argsort(importances)
plt.figure(figsize=(12,12))
plt.title('Feature Importances')
plt.barh(range(len(indices)), importances[indices], color='violet', align='center')
plt.yticks(range(len(indices)), [feature_names[i] for i in indices])
plt.xlabel('Relative Importance')
plt.show()
Stacking Model¶
Now, let's build a stacking model with the tuned models - decision tree, random forest and gradient boosting, then use XGBoost to get the final prediction.
In [ ]:
estimators=[('Decision Tree', dtree_tuned),('Random Forest', rf_tuned),
('Gradient Boosting', gb_tuned)]
final_estimator=XGBRegressor(random_state=1)
In [ ]:
stacking_estimator=StackingRegressor(estimators=estimators, final_estimator=final_estimator,cv=5)
stacking_estimator.fit(X_train,y_train)
In [ ]:
stacking_estimator_score=get_model_score(stacking_estimator)
Comparing all models¶
In [ ]:
# defining list of models
models = [dtree, dtree_tuned, rf_estimator, rf_tuned, ab_regressor, ab_tuned, gb_estimator, gb_tuned, xgb_estimator,
xgb_tuned, stacking_estimator]
# defining empty lists to add train and test results
r2_train = []
r2_test = []
rmse_train= []
rmse_test= []
# looping through all the models to get the rmse and r2 scores
for model in models:
# accuracy score
j = get_model_score(model,False)
r2_train.append(j[0])
r2_test.append(j[1])
rmse_train.append(j[2])
rmse_test.append(j[3])
In [ ]:
comparison_frame = pd.DataFrame({'Model':['Decision Tree','Tuned Decision Tree','Random Forest','Tuned Random Forest',
'AdaBoost Regressor', 'Tuned AdaBoost Regressor',
'Gradient Boosting Regressor', 'Tuned Gradient Boosting Regressor',
'XGBoost Regressor', 'Tuned XGBoost Regressor','Stacking Regressor'],
'Train_r2': r2_train,'Test_r2': r2_test,
'Train_RMSE':rmse_train,'Test_RMSE':rmse_test})
comparison_frame
In [ ]:
# So plot observed and predicted values of the test data for the best model i.e. tuned gradient boosting model
fig, ax = plt.subplots(figsize=(8, 6))
y_pred=gb_tuned.predict(X_test)
ax.scatter(y_test, y_pred, edgecolors=(0, 0, 1))
ax.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'k--', lw=3)
ax.set_xlabel('Observed')
ax.set_ylabel('Predicted')
ax.set_title("Observed vs Predicted")
plt.grid()
plt.show()
Business Recommendations¶
In [ ]: